Skip to content

feat(report): credit rule evidence beyond review comments - #699

Merged
theCodeDrift merged 7 commits into
mainfrom
report-rule-evidence
Nov 17, 2026
Merged

theCodeDrift merged 7 commits into
mainfrom
report-rule-evidence

Conversation

@theCodeDrift

Copy link
Copy Markdown
Member

The CLI half of rule evidence. tskl.es started accepting it in taskless/taskless#527, which is now deployed. Until now, a rule chosen from repeated fix-up commits, docs or tests published as "raised in 0 reviews", and a repository that pushes to main showed zero on every rule.

What changes

  • Schema. report-v1.schema.json is regenerated from the live tskl.es schema. It gains per-rule sources (seven kinds) and evidence (call-out counts per kind). Both are optional, and schemaVersion stays "1". The CLI never sent the October sources field either, so this adds both.
  • Facts. Every rule (rules, legacy, cloud) takes optional sources and evidence, which are carried to the payload unchanged. The kinds live in report/evidence.ts with the group of each. A test holds them equal to the vendored schema, so a refresh that adds a kind fails until it is given a group.
  • Checks. Before any network call, the CLI refuses what tskl.es's checkReport refuses, under REPORT_INVALID and naming the rule and the field:
    • sources on some rules but not all
    • a kind listed twice
    • reviews with no review comment
    • commits/issues/docs/tests listed without a count
    • a count for a kind the rule does not list, or evidence without sources
  • Summary. Each rule line carries the page's two totals, for example Rule a (Taskless ast-grep rule): 6 times in reviews, 1 time in conventions. They are summed and ordered the way the page does it.
  • Recipes.
    • onboard (topic v15) tallies call-outs from every source per candidate, by kind, while mining. Commits that made the fix count as commits evidence, never as surfaced.
    • report (topic v10) renames "Counting surfaced" to "Counting evidence" and defines one call-out per kind. Kinds and counts only, never paths.

Facts without the new fields produce the same payload as before.

Differences from the issue

The shipped service contract names the tracker kind issues, not trackers. TODO/FIXME comments count as conventions. The summary shows two group totals, not a single "strongest evidence". All of it follows taskless/taskless#527's design.

Verification

  • pnpm typecheck, pnpm lint, pnpm test (3348 passed).
  • Each of the six evidence checks was disabled in turn, and its test failed each time.
  • A trial archive kept every prior scenario of cli-report and cli-onboard and added 12 new ones.
  • The OpenSpec change report-rule-evidence is archived in this PR.

Post-merge: publish an onboarding report from a repository with no PR history and confirm its rules show "times in reviews" / "times in conventions" on the page.

Fixes #681

Regenerates the ReportV1 schema now that tskl.es credits commits, issues,
docs and tests (taskless/taskless#527), and opens the report-rule-evidence
change for the CLI half.

Refs #681
…ach kind

Onboard credits commits, issues, code comments, agent files, docs and
tests as their own kind of evidence instead of dropping everything but
review comments; the report recipe defines one call-out per kind.

Refs #681
@theCodeDrift

Copy link
Copy Markdown
Member Author

@claude /review

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @theCodeDrift's task in 2m 37s —— View job


Review: feat(report): credit rule evidence beyond review comments

Review mode: incremental — read 0 prior review thread(s) before reviewing. .prior-review.json contains no review threads, reviews, or top-level comments beyond the trigger comment itself, so this is the first review pass of this PR; the whole diff was assessed fresh.

Todo

  • Read prior review data (none found)
  • Fetch PR diff and file list
  • Review schema/type changes (report-v1.schema.json, types.ts, report-facts.ts)
  • Review evidence module (evidence.ts) and its validation checks
  • Review assemble.ts / summary.ts changes
  • Review agent docs (onboard.md, report.md) and OpenSpec specs
  • Review tests for adequacy
  • Post final summary

Summary

This is a clean, well-scoped implementation. I traced the data flow end to end:

  • report/evidence.ts defines the seven source kinds, their groups (conventions/reviews), and the cross-field validation (evidenceIssues) plus the summary math (groupTotals). The satisfies Record<string, SourceGroup> table and the schema-equality test (report.test.ts: "are the vendored schema's, each with a group") correctly guard against a schema refresh silently adding an ungrouped kind.
  • schemas/report-facts.ts builds sources/evidence zod schemas directly from RULE_SOURCES/EVIDENCE_SOURCES, so the facts schema and the kind table can't drift.
  • report/assemble.ts: evidenceOf() only adds sources/evidence keys when present (verified against the "adds neither field when the facts give neither" test), and checkEvidence() runs before enforceGate(), across all three tiers (rules, legacy, cloud) combined — matching design D3's "on every tier" rule. I manually re-derived each of the six refusal conditions against evidenceIssues() and they match the spec exactly (all-or-none across tiers, duplicate kind, reviews needs a review comment, counted kinds need a count, a count needs its kind, evidence needs sources).
  • report/summary.ts's evidenceSuffix(): I hand-checked the test fixtures in renderSummary (rule a: surfaced.reviewComments=2 + evidence.commits=4 → 6 reviews, evidence.docs=1 → 1 convention; rule d's tie breaks to reviews-first) — the sums and ordering match design D4 precisely.
  • The vendored report-v1.schema.json additions (sources enum, evidence properties with exclusiveMinimum: 0) line up with the zod facts schema's min(1) constraint.
  • Test coverage is thorough: each of the six checks has a dedicated failing case plus a passing counterexample, and the onboard/report recipe doc tests assert the specific wording changes.

I did not find any correctness, security, or logic bugs in the reviewed code.

One minor nit (cosmetic, not blocking):

  • packages/cli/src/agent/report.md:89 — the edited line ("today, what review raised over the window, the link, and when it expires. Relay it to the user as") runs noticeably longer (~102 chars) than the ~72-character wrap width used throughout the rest of the file, including the untouched lines immediately before and after it. Likely just a missed rewrap after editing; worth a pass to keep the file's wrapping consistent.

No concerns with the OpenSpec archive content (design.md/proposal.md/spec deltas) — they're documentation mirroring the shipped behavior and read consistently with the code.

@theCodeDrift

Copy link
Copy Markdown
Member Author

Re: @github-actions — "Claude finished @theCodeDrift's task in 2m 37s … One minor nit (cosmetic, not blocking): packages/cli/src/agent/report.md:89"
#699 (comment)

Rewrapped the "Show the result" step to the file's 72-column width in c9d0706; the rest of the edit had pushed the paragraph after it out of shape too, so the whole step is reflowed. No other findings to address.

— AI Coding Agent

@theCodeDrift
theCodeDrift merged commit 9ff3453 into main Nov 17, 2026
4 checks passed
@theCodeDrift
theCodeDrift deleted the report-rule-evidence branch November 17, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

report: credit rule evidence beyond review comments (commits, docs, tests, trackers)

1 participant